Processor-based systems, such as computer systems, use memory devices, such as dynamic random access memory (“DRAM”) devices, as system memory to store instructions and data that are accessed by a processor. In a typical computer system, the processor communicates with the system memory through a processor bus and a memory controller. The processor issues a memory request, which includes a memory command, such as a read command, and an address designating the location from which data or instructions are to be read or to which data or instructions are to be written. The memory controller uses the command and address to generate appropriate command signals as well as row and column addresses, which are applied to the system memory. In response to the commands and addresses, data is transferred between the system memory and the processor. The memory controller is often part of a system controller, which also includes bus bridge circuitry for coupling the processor bus to an expansion bus, such as a PCI bus.
Although the operating speed of memory devices has continuously increased, this increase in operating speed has not kept pace with increases in the operating speed of processors. Even slower has been the increase speed at which memory commands, addresses and data can be coupled between processors and memory devices. The relatively slow speed of memory controllers and memory devices limits the data bandwidth between the processor and the memory devices.
One approach to increasing the data bandwidth to and from memory devices is to use multiple memory devices coupled to the processor through a memory hub as shown in FIG. 1. A computer system 100 using a memory hub architecture includes a processor 104 for performing various computing functions, such as executing specific software to perform specific calculations or tasks. The processor 104 includes a processor bus 106 that normally includes an address bus, a control bus, and a data bus. The processor bus 106 is typically coupled to cache memory 108, which, is typically static random access memory (“SRAM”). Finally, the processor bus 106 is coupled to a system controller 110, which is also sometimes referred to as a bus bridge.
The system controller 110 contains a memory hub controller 128 that is coupled to the processor 104. The memory hub controller 128 is also coupled to several memory modules 130a-n through a bus system 134. Each of the memory modules 130a-n includes a memory hub 140 coupled to several memory devices 148 through command, address and data buses, collectively shown as bus 150. The memory hub 140 efficiently routes memory requests and responses between the controller 128 and the memory devices 148. Computer systems employing this architecture can have a higher bandwidth because the processor 104 can access one memory module 130a-n while another memory module 130a-n is responding to a prior memory access. For example, the processor 104 can output write data to one of the memory modules 130a-n in the system while another memory module 130a-n in the system is preparing to provide read data to the processor 104. The operating efficiency of computer systems using a memory hub architecture can make it more practical to vastly increase data bandwidth of a memory system. A memory hub architecture can also provide greatly increased memory capacity in computer systems.
The system controller 110 also serves as a communications path to the processor 104 for a variety of other components. More specifically, the system controller 110 includes a graphics port that is typically coupled to a graphics controller 112, which is, in turn, coupled to a video terminal 114. The system controller 110 is also coupled to one or more input devices 118, such as a keyboard or a mouse, to allow an operator to interface with the computer system 100. Typically, the computer system 100 also includes one or more output devices 120, such as a printer, coupled to the processor 104 through the system controller 110. One or more data storage devices 124 are also typically coupled to the processor 104 through the system controller 110 to allow the processor 104 to store data or retrieve data from internal or external storage media (not shown). Examples of typical storage devices 124 include hard and floppy disks, tape cassettes, and compact disk read-only memories (CD-ROMs).
A memory hub architecture can greatly increase the rate at which data can be stored in and retrieved from memory because the point-to-point nature of the bus system 134 allows faster electrical signaling, and because the isolated nature of the DRAMs allows parallel or overlapping activity in several modules. In fact, a memory system using several memory modules each containing a memory hub can collectively transmit and receive data at such a high rate that the bus system 134 can become the “bottleneck” limiting the data bandwidth of the memory system.
One technique that has been used to maximize the data bandwidth of memory systems using a memory hub architecture is to implement the bus system 134 using a high-speed “downstream” bus 154 and an “upstream” bus 156 that is separate from the downstream bus 154. The downstream bus 154 and the upstream bus have the same width, i.e., number of conductors. The high-speed downstream bus 154 couples packets containing a memory command, an address, and write data from the memory hub controller 128 to the memory modules 130 and from the memory modules 130 to memory modules 130 located further away from the memory hub controller 128. The high-speed upstream bus 156 couples a packet containing read data and an identifier from memory modules 130 to the memory hub controller 128 and from the memory modules 130 to memory modules 130 located closer to the memory hub controller 128. The use of two different uni-directional buses has the advantage of eliminating turn around latency and allowing the memory hub controller to simultaneously transmit and receive data.
The data bandwidth of memory systems using a memory hub architecture can also be maximized by using a different type of memory signaling protocol. Rather than using traditional address, data and control buses, the address, data and control bits for each memory request or “transaction” are sent together in a single packet. The packet includes a command header followed by read or write data. The command header includes bits corresponding to a memory command, such as a write or a read command, identifying bits that specify the memory module to which the request is directed, and address bits that specify the address of the memory devices 148 in the specified memory module that is being accessed with the request. The command header may also specify the quantity of read or write data that follows the command header. The use of a packetized memory system allows the memory hub controller 128 to issue a memory request by simply transmitting a packet instead of transmitting a sequence of command, address and, in the case of a write request, write data signals. As a result, the memory hub controller 128 can issue memory requests at a faster rate. Furthermore, a packetized memory system frees the memory hub controller 128 from having to keep track of the processing details of each memory request. Instead, the memory hub controller 128 need only transmit the packet. The memory hub 140 in the memory module 130 to which the memory request is directed then processes the memory request without further interaction with the memory hub controller 128. In the case of a read request, the memory hub 140 transmits a packet back to the memory hub controller 128, either directly or through intervening memory modules 130, that contains the read data as well as identifying bits in a command header identifying the read data. The memory hub controller 128 uses the identifying bits to associate the read data with a specific memory request.
Although the memory hub architecture shown in FIG. 1 can provide a significant increase in the data bandwidth between the processor 104 and the memory devices 148, it nevertheless can provide at less than optimum performance. In particular, the capacity of either the downstream bus 154 or the upstream bus 156 can limit the rate at which packets can be coupled to or from the memory modules 130, thereby limiting data bandwidth. The particular bus 154, 156 that becomes overloaded normally depends on the nature of the memory accesses. Memory writes result in a significantly greater amount of traffic through the downstream bus 154 compared to the traffic through the upstream bus 156. Memory reads still require that packets containing commands and addresses be coupled through the downstream bus 154, but they generally result in significantly more traffic being routed through the upstream bus 156. Therefore, memory accesses primarily consisting of writes tend to overload the downstream bus 154, and memory accesses consisting primarily of reads tend to overload the upstream bus 156. In either case, the result is a reduced data bandwidth between the processor 104 and the memory devices 148.
There is therefore a need for a memory hub architecture that is more able to avoid overloading either the downstream bus or the upstream bus coupled between a memory hub controller and one or more memory hubs.